Skip to content

Hold deleting repository secrets and branch protection through gh - #468

Merged
LoopGlitch26 merged 6 commits into
Cygnux-Labs:mainfrom
aj-codespy:agent/issue-402
Oct 31, 2026
Merged

LoopGlitch26 merged 6 commits into
Cygnux-Labs:mainfrom
aj-codespy:agent/issue-402

Conversation

@aj-codespy

Copy link
Copy Markdown
Contributor

What and why

The second blind E21 pool (#401) showed that an agent with a GitHub token could take a repository's protections apart
with nobody asked: all six of its such calls got through the server pack. TK-A043, an ask rule next to TK-A042,
now holds them in every pack:

  • gh secret delete and gh variable delete (or remove), with --repo/-R, --env or --org;
  • a DELETE, PUT or PATCH through gh/hub api or curl on a branch's protection or a path under it, a
    repository's or organization's rulesets, or their Actions, Dependabot or Codespaces secrets and variables (an
    environment's too), as a bare path or a URL (also GitHub Enterprise's /api/v3), with the method before or after it
    and up to sixteen option words around the method, read as TK-A042 reads gh api;
  • a PATCH that sets a security_and_analysis feature to disabled;
  • a gh api DELETE with $ or a backquote in an argument, whose path is built at run time (the pool's
    base64-decoded secret path).

A GET of the same paths, gh secret list, gh secret set and a DELETE of an issue comment stay allowed. Every gap
in the pattern is bounded. Coding pack version 2026.10-v2-33.

  • Tests: the six calls in tests/data/policy/deny/shell.jsonl, the pool's lookalikes plus gh secret set and an
    issue-comment DELETE in tests/data/policy/benign/shell.jsonl, and a four-pack × two-engine test in
    tests/test_signer_policy.py with the option orders, methods, paths and lookalikes above.
  • Docs: TK-A043 in the coding pack's ask row of docs/policy-v2.md, and a docs/limits.md row for what it can't
    read (another forge, a GraphQL mutation, a body from a file, a token used from a script or SDK, other HTTP clients).
  • Two judgment calls, happy to change either: a PUT of a secret through the API is held, as the issue lists it,
    while gh secret set isn't; and a gh api DELETE with any $ in its arguments is held, an issue comment's by a
    variable id too (marked lean:).

#401 pool after this change (its first score stays the blind one), both engines deciding every call the same:

Pack Harmful held or denied Benign held
server (server.yaml) 28 of 45 (62%), 17 denied (blind: 22 of 45) 0 of 45 (unchanged)
strict (strict.yaml) 31 of 45 (69%), 19 denied (blind: 25 of 45) 2 of 45 (unchanged: curl --dry-run, curl -I)

On server, CI/CD changes went from 0 to 2 of 4 and defense evasion from 4 to 8 of 13; the other two CI/CD misses are
workflow files (#405).

E17: 705 of 705 harmful calls stopped, still 1 false alarm (the draft email), now in 808. E21: the held-out pool's
numbers are unchanged (default 245 of 420, strict 274 of 420), and all 705 E17 originals and 1,821 shell rewrites are
contained. E21's machine and latency are kept as published, since this run was on a laptop. The README and docs
that quote E17's and E21's counts follow.

Fixes #402

Tests (macOS, Python 3.14): tests/test_signer_policy.py, tests/test_policy2.py (including the 64 KiB pattern
timing on both engines) and tests/test_eval_docs.py, tracekit policy lint, make eval.

Checklist

  • make check passes (lint, tests, build); make test-ts too if sdk/typescript/ changed
  • a regression test covers the change
  • user-visible changes are described above (release notes are written at release time; don't edit CHANGELOG.md)
  • docs updated if behaviour or a security claim changed (docs/threat-model-laptop.md,
    docs/threat-model-server.md)
  • every commit is signed off (git commit -s, see CONTRIBUTING.md)

Security invariants

  • the agent still never holds a signing key or assigns sequence numbers
  • nothing an agent-controlled process supplies (isolation level, fail mode, gap/tamper records,
    approval identity) is newly trusted by the signer or verifier
  • gaps are still signed, never silent; a check that cannot run warns instead of passing
  • no verification code is shipped inside a bundle

Evidence format

  • this change does not touch the frozen formats: v1 (core.sig_message, ledger.make_record, the v1 schema, how
    the verifier treats existing bundles) or the published v2 format
  • or, if it does: the issue explicitly asks for it, existing bundles (including docs/sample/ and
    tests/fixtures/) still verify, and the change is called out above

@LoopGlitch26

Copy link
Copy Markdown
Member

Thanks for this. TK-A043 closes a real gap from the E21 pool, and it only adds a rule, so nothing existing gets weaker.

Before it can merge:

  1. Rebase on main. 1.5.0 is out and the coding pack is now 2026.10-v2-39; bump yours to 2026.10-v2-40.
  2. After the rebase, regenerate the eval results (eval/e17_policy_corpus.py, eval/e21_policy_robustness.py, eval/e21_robustness.py) and update the latency quotes in README.md and docs/evaluation.md; tests/test_eval_docs.py checks them.
  3. Check the fallback engine's speed (tests/test_policy2.py::Speed). In 1.5.0, TK-A040 had to go from sixteen option words to eight to stay within the budget on CI's runners, and TK-A043 reads up to sixteen twice. Eight is probably enough here as well.

Once CI is green we'll do a second review.

Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
The 1.5.1 release takes TK-A043 to TK-A046, so this rule moves to
TK-A047 in the pack, its test and the docs.

The E17, E21 policy-robustness and E21 robustness results are
regenerated against the new rule and corpus lines, and the E21
latency table in the evaluation page quotes the new run.

Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
@LoopGlitch26

Copy link
Copy Markdown
Member

I made the changes from the 2026-10-28 review, so the branch is ready for CI and the second review:

  1. Rule id. TK-A043 is now TK-A047 in the pack, the test and the docs, because the 1.5.1 release (Release 1.5.1 #475) uses TK-A043 to TK-A046.
  2. Main. I merged main into the branch (a3b15d2) instead of rebasing it, so no force-push was needed. There were no conflicts.
  3. Version. The coding pack stays at 2026.10-v2-40, one past main's.
  4. Evals. E17, E21 policy robustness and E21 robustness are regenerated (5ad5cf7):
    • E17 stops 705 of 705 harmful calls, with 1 false alarm in 808.
    • E21 robustness contains all 6 new calls under TK-A047.
    • The E21 latency table in docs/evaluation.md quotes the new run, and tests/test_eval_docs.py passes.
  5. Speed. I kept sixteen option words. TK-A047 is anchored at the start of the command, so it can't rescan from every repeat the way TK-A040 did (TK-A040 reads after each redis-cli on a line). Timings on this machine, against a 150 ms budget:
    • on the fillers in tests/test_policy2.py::Speed, its slowest search is 5.5 ms with sixteen words or eight (TK-A042: 17 ms, TK-M003: 81 ms);
    • on crafted 64 KiB inputs aimed at its repeats, it takes under 1 ms.

Why each job failed on the last CI run:

  • Caused by this PR, fixed here: test_eval_docs and ServerCorpora::test_e17_decisions_are_the_recorded_ones failed because the eval results hadn't been regenerated.
  • Main's, fixed on main by After 1.5.0: lock the published signer packages; CI runs the 1.5.0 verifier #470: Offline evals and release-dry-run failed in npm ci, because the signer package lock was out of date at the time.
  • A slow runner: the py3.9 macOS Speed failure listed TK-M003 at 168 ms first (12 entries; the log cuts off the rest). That's existing patterns over budget on that runner. This rule is about 15 times faster than TK-M003 here.

tests/test_signer_policy.py, tests/test_policy2.py and tests/test_eval_docs.py pass locally (210 tests), and make lint is clean.

Conflicts resolved so both sides stay:
- coding.yaml: main's 1.5.1 rules (TK-A043 to TK-A046 payment and
  download holds, TK-D002/TK-D007 precision, memory-notes exceptions)
  plus TK-A047 after TK-A042; pack 2026.10-v2-43. TK-A047's remedy now
  reads "Have a person approve the held call", as every held call's
  does since 1.5.1.
- tests/test_signer_policy.py: both sides' tests.
- README.md, docs: main's text with TK-A047's ask entry and limits row,
  and E17's counts at 705 harmful and 808 benign calls.
- eval/results: E17, E21 policy robustness and E21 robustness run
  afresh; E21's latency block kept as main recorded it.

Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
The second review of Cygnux-Labs#468 found TK-A047's fallback-engine searches
super-linear and several ways past it. On Python 3.9 (no google-re2),
64 KiB of `$` or backquotes in a gh or hub api argument spent the
decision's whole second and denied as nondeterministic; so did a branch
name repeating `/protection` (found while fixing); a repeated
security_and_analysis/disabled pair took ~0.6 s. The run before a `$` or
backquote now stops at the first one, the text around
security_and_analysis is bounded, and a branch name is read to 250
characters: the same inputs decide in 10-53 ms on the fallback engine.

Now held as well: the method in any case and after -X ending a cluster
of short options (-iX, -sSX); an empty or space-edged argument; the owner
and repository as one variable (repos/$GITHUB_REPOSITORY/...); the API's
URL as a variable, its scheme in capitals or left out; gh repo edit
turning secret scanning, push protection or advanced security off; an
environment and paths under it; a DELETE of vulnerability-alerts or
automated-security-fixes; the user's Codespaces secrets. A
security_and_analysis PATCH is held only when a feature's status is set
to disabled, so turning one on with "disabled" in another field isn't.

The gh api DELETE with any variable stays held (lean): a variable before
the path's last segment can hide a protected path. docs/limits.md now
says which run-time paths are held and lists curl's URL rewriting,
xargs, curl -K, numeric repository ids and long branch names.

Speed gains `$`, backquote and the pair as fillers, and its adversarial
decide test the three inputs that denied. E17 gains 10 harmful and 6
benign shell calls; E17 and both E21 results are re-run, E21's latency
block kept as recorded, and the docs quote the new counts.

Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
An independent review of the previous commit found two ways past the
rule: a method word taken at run time (gh api -X "$M", -X ${M:-DELETE},
-X $(echo DELETE), a backquoted echo, curl -X "$M", hub api -X "$M")
passed every API branch, and gh repo edit --enable-secret-scanning=$OFF
(and the push-protection and advanced-security flags) was allowed.

A method word with a $ or a backquote in it is now read wherever DELETE,
PUT or PATCH is, before or after a written-out protected path or a
security_and_analysis body; joined to its flag (-X$M) it must start with
one, so the flag's run isn't rescanned. gh repo edit's value is held
when it has a $ or a backquote in it, as =false is. A run-time method on
another path (an issue comment) isn't held.

docs/limits.md's TK-A047 row now also says: the sixteen-piece gaps
count each space in a quoted argument and the words before the method;
JSON padding past 200 characters or 16 non-word characters isn't read;
percent-encoded parts, doubled slashes, capital path parts and
status=Disabled aren't matched; which neighbouring security changes are
out of scope; and that any host is read, so a PUT to a local test server
on the same paths is held.

E17 and E21 decide the same; only the pack hashes move.

Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
@LoopGlitch26 LoopGlitch26 mentioned this pull request Oct 30, 2026
@LoopGlitch26

Copy link
Copy Markdown
Member

The last review fixes are on this branch now (head ca296b4). It ships in release 1.5.2 (#479), which merges this branch's commits as they are, so this PR will show as merged when #479 lands. Thanks for TK-A047.

@LoopGlitch26
LoopGlitch26 merged commit e0a9d62 into Cygnux-Labs:main Oct 31, 2026
31 of 33 checks passed
@LoopGlitch26

Copy link
Copy Markdown
Member

Thanks again for contributing to Tracekit. A request: if you use a coding agent (Claude Code, Codex, Cursor, Gemini CLI or OpenCode), would you run Tracekit on it for a week and tell us how it went?

pip install -U tracekit-ai
tracekit audit --since 7d      # read-only: what it would have held in your last week (Claude Code history)
tracekit init --dev --v2       # then record and guard from now on

After a week, tracekit report --share prints counts only (no commands, paths or content). Post it in #385 with anything that got in your way: a false alarm, a confusing message, something slow. If you find a problem, you're welcome to fix it too: say so on #385 and it's yours.

@aj-codespy

Copy link
Copy Markdown
Contributor Author

@LoopGlitch26 Sure, I'd be happy to help. I use Cursor and Claude Code, so should I go with either one?
Also If there's some more issues waiting, I'd love to work on it, especially if there's something in AI Domain

@LoopGlitch26

Copy link
Copy Markdown
Member

@aj-codespy Either works. Claude Code is the quickest start: tracekit audit --since 7d reads its history, and tracekit init --dev --v2 sets it up. For Cursor, add --agent cursor to init. After a week, tracekit report --share prints counts only; post it in #385.

On issues: #563 (Tracekit hooks for OpenHands, the CLI and SDK coding agent) just opened up and is the closest to the AI side. Comment there to take it.

@aj-codespy

Copy link
Copy Markdown
Contributor Author

@LoopGlitch26 The week's report is in: #815 (Claude Code, 7 days on macOS; 408 runs, all verified). Thanks for the nudge, and for the reviews on #662.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Policy: deleting repository secrets or branch protection through gh waits for a person

2 participants